Chen et al., 2021], and come up with new problems that drive innovation in art and science [Gromov,

Chen et al., 2021], and come up with new problems that drive innovation in art and science [Gromov,

This repo contains a dataset of Python programming puzzles which can be used to teach and evaluate an AI's programming proficiency. We present code generated by OpenAI's recently released codex 12-billion parameter neural network solving many of these puzzles. We hope this dataset will grow rapidly, and it is already diverse in terms of problem difficulty, domain, and algorithmic tools needed to solve the problems. Please propose a new puzzle or browse newly proposed puzzles or contribute through pull requests. To reproduce the results in the paper, see the solvers folder.

Programming competition problems are pervasive in the AI community. They can be used to evaluate programmers' abilities to solve artificial tasks as well as to test the limits of state-of-the-art algorithms. A research team from MIT, Allen Institute for AI and Microsoft Research recently introduced Python Programming Puzzles (P3), a novel and open-source collection of programming challenges that capture the essence of puzzles and can be used to teach and evaluate an AI's programming proficiency. The proposed puzzles take the form of a Python function with the answer as an argument. The goal is to find an input x that makes the output of the function true, i.e., a valid answer x satisfies f(x) True.